ED 200 899 CS 005 98* 

AOTHOB Hayfordr Paul Dip 

TIILl k Basic Program Analysis and Monitoring System in 

Raadingi Alternative Monitoring DeviGes* 
PUB DATE Bar 79 

llorE 22p, I paper presented at the Annual Meeting of the 

Affierican Sduaatlonal ReeearGh AssDciation (San 
FranaiacOi CA, April 8-12, 1979). 

IDBS PEICl MF01/PC01 Plus Postage, 

DEJCEIPTOIS Elementary Educatieni ^Program ivaluationi Reading 

AchieyeBent I *Reading Comprehension i Beading 
Diagnosisi *fieadlng Programsi ^Reading Ressarch: 
♦Reading Testsi Test Construction; Test Itsme; ^^rsst 
Validity 

IDINTIFIEES •program Analyiis and Monitoring in Reading 



ABSTSACT 

The Program Analysis and Monitoring in Beaaing (PAMl 
package helps teachers and administrators anlayze, understand^ and 
improve school reading programs by providing them with monitor 
reports and program reports* Both types of reports include results of 
student performance on tests of reading comprehenslon-'the monitor 
report provides detailed information (two types of scores, passage 
aeores and a monitor score) for each student in a reading Dlassi the 
program report lists more feneral inforaatlon, such as the average 
class monitor scores for each test given in a school year* During itg 
two years of development, the PAM package has used three types o£ 
multiple choice item forms to assess literal comprehenoioni a 
modified close measure^ traditional HH- detail verbatim Ltams, aad 
lOdified paraphrase items. The modified paraphrase items fallow the 
HB- detail format while substituting, whenever poisible, synonymous 
words or phrases for the content words in the passage sentance* The 
PAM package contains 2tt achievement monitors*-^ parallel fDrms at 
each of six levels of difficulty* Because of this, the test form a 
student takes is tailored to his or her ability level and he or ihe 
never takes the same form twice in a year* CAppendlaces contain 
information on interpretation of the scores and provide rules for 
constructing paraphrase items*! (MKM) 
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In their presentations on PAM— Program Analysis and Monitoring in 
Reading" Steven Kidder and Robert Ambrosino have described how the 
Package helps administrators and teachers analy^a, underscandj and improve 
school reading programs by providing them with Monitor Reports and Program 
Reports. The utility of- both types of reports depends on the achievement 
monitors used to measure pupil progress in reading* Both reports include 
results of pupil perfomance on testJ of reading comprehension* The Monitor 
Report provides very detailed information^' two types of scores ^ passage 
icores and a Monitor Score— for each pupil in a reading class* The passage 
scores are the percentage of items correct on each test passage* The Monitor 
Score is a conservative estimate of the most difficult or complex prose the 
pupil can literally comprehend. It is based on the pupil *s pattern of 
performance on the test passages* The Program Report lists more general 
ln£orMtlon«-average class monitor scores for each test aininlstration during 
a school year* 

The properties of the Achievement Monitors used in the P^ Package 
permit estimates of progress in literal comprehension for Individuals and 
groups both throughout a school year and from year to year* These properties 
are the validity of the tests measures of literal comprehension, the 
ranking of test passages on a scale of difficulty from 1 to 26, sensitivity 
to gains over periods as short as ten weeks, and a high ceiling preventing 
all but the most extraordinary elementary*school pupils from topping out- 

Durlng its two years of development, the PAM. Package has used three 
types of multiple-choice itan forms to assess literal cmnprehension. In 
1977*78, P^ used a modified clo^e measure, the Multiple-Choice Cloze, in 
its achievement monitors* In 1978-79 PAM ^s switched to t^^o other types 
©£ items, traditional WH* detail verbatim ItOTS and modified paraphrase 
items* .The modified paraphrase items follow the WH* detail format while 
substituting wherever possible synonymous words or phrases for the content 



words (nounSj verbSj adjectives^ and adverbs) in the passage sentence. 

The three Item foms which have been used in the P^I achievement monitors 

are rule-based. All three have been validated in studies reported elsewhere 

(O'Reilly & Streeter, 1977| KLdder, Hayfordp & Salter, 1977i toyford & 

Salter^ 1978), and all eKlst in large batteries of passages scaled by 

difficulty through the use of readability formulas, ^ (Illustrations of each- - 

it«a type appear in Figures 1, 2, and 3.) 

Ttoe does not permit a thorough description of all three types of 

items. I will try here to focus my discussion on the paraphrase item forrrif 

which seems to hold the TOSt promise for Vm aGhievenent monitoring. During 

1978-79, the PAM Achievement Monitors have comprised 24 test forms with a 

total of 720 tt^s, 90 percent paraphrase items and 10 percent WH- verbatim 

ItCTis. PAH switched to paraphrase itms because of their superior theoretical 

justification as measures of literal comprehension (Anderson, 1972). Though 

studies involving the Multiple-Choice Close and the verbatim WH- detail item 

2 

have yielded very high reliability and validity coef flcientis both these 
it«n forms share the weakness of being susceptible to test-wlseness or test- 
taking skills. Ctothographic matching, for eKample, can »«solve" a WH- 
verbatim item, and general graimnatical and sCTiantic skills can be ©nployed 
to get rlsht answers on Khiltiple-Choice Clo^e itens in the absence of 
understandins of specific context. It should be noted that the verbatim 

''Test batteries include 1,725 Multiple-Choice Cloze exercises, 300 
passages with 8 WH- verbattai Itrais each, and 124 passages with 6 paraphrase 
items each. 

^For eKarapli, the Multiple-Choice Cloze and the WH- verbatim itmis have 
correlated in the high .60* s and the .70* s with the California Achievement 
Test in grades 1-9, and the Multiple-Choice Cloze has correlated in the high 
.70 »i and low ^80 »s with the Gates-^cGlnitie and the Stanford Achievement 
Test in a highly generalizable sample in grades 3, 6* and 9* (Incidentally^ 
the paraphrase Item form had similar correlations with the Gates-MacGlnltle 
and with the Hultlple-Choice Close.) See O'Reilly & Streeter, 1977, and 
Ittdder, Hayford, & Salter, 1977. 
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Though Loki, the fire god, was ^ and ready- 
witted, his 12 was really evil. He was, indeed, 
the 13 of most of the 14 — - which befell 

the gods. He was _.JJ= in trouble, yet often 

16 because the gods valued his . It 

was he who _ — 
so that for a 
do without him, 



18 



ways out of 19 for them, 
20 time they felt that they could not 
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13^ 



0 




15^ 



1. handsome 

2. alcoholic 

3. fragrant 
4* squeaky 

5. fundamental 

1. pound 

2. emblem 

3. awl 

4. nature 
S* uniform 

1. salvage 

2. cause 

3. stress 
4* sphere 

5* protection 

L applications 

2. nominations 
3i scratches 

4. misfortunes 

5. mosquitoes 

1, scarcely 

a, precisely 

3. strangely 

4. immensely 
5« constantly 
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forgiven 
2* postponed 

3. quadrupled 

4. puzzled 
5* admitted 

1. admittance 

2. discomfort 
3* cleverness 
4, propeller 
Si^plasma 

1^ silenced 
2* afflrmed 

3. supported 
4^ froze 

5« found 

1, difficulty 
2* suspension 

3, conscience 

4. intermission 
5* desertion 

1, swampy 

2, radiant 
3* long 

4. leak^ 

5. crisp 



Figure 1« Multiple-Choice Cloze Exercise. 
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One warm day In September, Roy went for a boat ride. He went 
with hla teacher and the class. They went on the big l^e by their 
sebDol* It was cool on the water. The wind was high. It made waves 
on the lake. All the children had fun that day* They enjoyed being 
nmmv the water on such a warm day. 



14# When did Roy go for a boat rids? 

Am in September 
B. in December 
Qm on Saturday 

15* Who did Roy go with? 

A# his brother and his sister 
B. his nether and his father 
C« his teacher and the class 

16. What was hight 

A. the water 

B. the wind 

C. tikm wave 

17. yho had fun timt day? 

A. ail the children 

B. the boys 

C. all the teachers 

la* Where did the children enjoy being? 

A. near the water 
B« in the classroom 
C. n^r the school 



Figure 2« PAM Achievement Monitor passage with WH- verba t to Itsns. 
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1 went into anoaier roonip where the walls and ceiling were all hung 
wound with cobwebs, except a naKOw pasiage for the artist to go in and 
Out# At entrance he called aloud to me not to disturb his webs. He 
lamanted the fatal mistake^ the world Md been so long in of using silk- 
womSf whils we had such plenty of domestic insects, who infinitely 
excelled the fomer, because they understood how to weave as well as spin. 
And he proposed farther, that by esploying spiders the clarge of dyeing 
silks would be wholly saved, whereof I was fully convinced when he showed 
me a vast number of flies most beautifully colored, wherewith he fed 
his spiderSf assuring me that the webs would take on their color* tod 
as he had them of all hues, ha hoped to fit every body's fancy, as soon 
as he could find proper food for the flies, of certain gums, oils, and 
other sticky matter, to give a strength and consistence to the threads. 



13, TOiat did 1 enter? 

A« ano titer cave 

B* another passaga 

G« another chamber . "'^'^ 

D« another building 

14« What were thm walls and ceiling completely covered with? 

A# dust balls 

Btf ' splderwebs 

Cm filshnati 

D* silk cloth 

i5« What was the slender corridor for? 

A« the Introduction of the spiders 

B« the artist's entrance and exit 

G« the artist to observe and e^erment 

D* the beautifully colored flies 



Figure 3* PAbI Achlevenent ^nltor passage with paraphrase Items. 
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16« When did the artist cry out to me not to upset his spiderwebs? 

iU whan X l^ft 

B« when I c^s in 

C# when 1 was fully convineed 

D« when I had plenty of domestic insects 

17. What did the artist deplore? 

A« the excessive numbers of spiders in the world 

B. the fatal mistake of using flies as food for silkworms 

Cw the mistake the world had been so long in of using weavers 

D* the world's longstanding^ deadly error of snploylng 
sll^orms 

18a What would be totally avoided by the us^e of spiderst 

Ac the cost of coloring silk cloth 

B« the expense of cultivating silfct^orms 

0# the charge of dyeing flies 

D* the cost of killing insects 



Figure 3. (Continued) 
.6- g 



Items only appear on test forms containing paasages at the lowest difficulty 
levela«l to 6| since only pupils just beginning to read respond to these 
ItffliSy there seoas little danger of their responses being contaminated by 
tes t-wlsenes s • 

The development of the test items used in PAM achievement monitoring 
has stressed validity^ especially construct validity* This seems often to 
have been neglected In the developMnt of achievement tests In readings 
perhaps in the belief that no theoretical rationale is necessary for items 
which do, after all, involve responding to the printed word. Literal 
comprehension has been enphasized because it is the goal, the major objec- 
tive, of elOTentary reading programs. All the detailed, minute Instructional 
objectives of elementary reading programs are but means to the end of literal 
comprehensions under standing the explicit meaning of the written word* 
Further, literal comprehension is also the foundation for all other compre- 
^ henslonf whether it be literary prose, poetry, or the tortured polemics of 
the psycholinguists* Concern for the validity of the PAH achiev©rient 
monitors has been warranted| the reports provided to teachers and admnls- 
trators would have little practical utility for reading progrmns if the 
information they contained were based on tests which measured something other 
than or in addition to reading comprehenslonr 

The PAM Package contains 24 achieVOTient monitors, 4 parallel forms at 
each of six levels overlapping in difficulty (Appendix A)* Because the 
passages on each form are scaled by difficulty and there are 6 leveli of 
achievement monitors, the test form a pupil takes is tailored to his level 
of ability and he never takes the same form twice in a year. It is conceivable, 
for example, that a large heterogeneous fifth-* or sixth-grade cla^^s could 
see pupils taking all 24 of the achlev^ent monitors at a given test atolnls- 
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tration. To be able to follow pupil progress from grade 2 through grade 6 
or higher on the same scale of prose difficulty requires accurate scaling 
of the passages on the achievement monitors^ In most cases, pupil perfor- 
mance on the test forms has confirmed the accuracy of the readability scores 
used to scale the passages. But there are several eases of reversals" that is, 
where groups perforrA better on a passage of supposedly higher difficulty 
than on a lower-difficulty passage* Practically speaking, for an individual 
pupil this occasional problem of passage scaling is taken into considera- 
tion by the algorithm used to assign a Monitor Score* If a pupil *s perfor- 
mance is erratic (whether through his own inconsistency or through a problCTi 
in passage scaling), his score is adjusted to avoid inflation* (The three 
basic niles for Monitor Score assignment are explained in AppendlK A*) I 
will return later to some recoimendatlons for ImproviT^ passage scaling* 

A distinct advantage of the achlevOTient monitors is their sensitivit 
to change over ten-week periods* If group performances did not show change 
over short periods, the achievement monitors would lose much of their utility 
for the teacher, who in most cases already has access to Information based on 
wide-interval testing* In 1977-78, when the Multiple-Choice Cloze was used 
in PAM, grade 2 pupils averaged increases of *78 points on the 1 to 26 scale 
of difficulty over ten-week lntervals» and grade 5 pupils averaged *42 points 
This year, over three test aininistrations using the paraphrase item form, 
scores for pupils in grades 2, 3, 5^ and 6 have averaged the following in- 
creases, respectively, over ten-week Intervals! *49, *4l, « 28^ and #11* 
These increases have occurred despite a significant departure from last year^ 
testing procedures. Last year a screening test was initially administered 
which placed pupils rather accurately for the first achlevraient monitor 
ainlnistration. This year the screening test was not administered. 
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the first test administration thus Involving some mislavelllng of pupils 
and apparently deflating increases from TAl to TA 2. 

A final advantage of the paraphrase items used in is their high ceil- 
ing • For example, 8 pupil i out of some 4500 have topped out on the highest 
level (level 6) PAH monitors. Theie pupils were sixth-graders. Last yearf 
in contrast, with the cloze items, at least five times as many pupils (fifth- 
graders) from a sample half as large topped out at level 6# What this sug- 
gests is the potential of extending PAM into junior high school, and even 
into senior high for students with reading problems. Indeed, one adminis- 
trator is planning next year to follow his seventh-grade students who are 
In compensatory programs* 

The experience of developing PAM and its paraphrase items has resulted 
In the acquisition of practical information which may be worth conveying* 
The observations which follow are related, respectively, to iten writing, 
validity and reliability, and passage scaling* 

The advantages of writing items to rules (AppendlK B) are standardiza- 
tion and elimnation of subjectivity and idiosyncratic intepretation* But 
writing paraphrase Items is not staple* It Involves three problems* One is 
that not all sentences^ clauses, phrases, etc* can be paraphrased* Not all 
sentences will yield items (and awkwardness and barbarousness should at all 
costs be avoided)* A second is that it is hard to control vocabulary 
difficulty when paraphrasing* We have tried to do this by using graded 
vocabulary lists, but it will not always work* Hard sentences usually 
result in hard paraphrases* The third problem is related to the artfulness 
involved in item writing. In this case the art Is not involved in concocting 
ItmSf it is involved in identifying sentences which will yield items (and 
having the a' llity quickly to recognlie a sentence destined to remin fruit- 
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less). It should also be noted that a staff endeavoring to produGe large 
batteries of such Itens (or any similar items) must include someone with 
high-level editorial skill s« 

The three item types discussed in this paper have been employed in 
large-scale validation studies. Correlational studies have discovered 
high reliability and validity coefficients. But the relevance of such 
traditional estiiMtes of reliability and validity is questionable for test 
which are used as the PAM monitors are* For Instance^ in the validation 
studies samples were generaliEable and randomly selected and tests were 
constructed for the sake of yielding maxtoum variance* But In practice 
both the range of passage difficulty in test forms and the range of pupil 
ability are greatly restricted, MsOj total test scores are not used in 
FAM, and the monitor scores which are used involve another restriction 
affecting variance. The range of possible Monitor Scores on a given test 
form is about 10 points^ whereas a total test score range could be 30 
points* 

Four suggestions for improving scaling are proposed* (1) Revise Item 
further* (2) Rescale passages using Easch analysis to produce a new, 
Rasch scale. (3) Rescale passages using ©apirical difficulty (a bit of a 
problem here, because the difficulty levels would come Into question)* 
(4) Produce a new battery of passages with (a) a simpler scale, say, 1 to 
13, and with (b) clearer gaps (in readability scores) between passages on 
the scale. 
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PAM--PROGRAM ANALYSIS AND MONITORING IN REAniNG 

Monitor Reports and Monitor Scores 

pypil performance on the Literal Comprehension Achievement Monitors 
used in the PAR ^oject is reported by reading class after each test 
administration* The Infonnation in the leveral columns of the computerized 
Monitor Report and the determination of monitor scores are described below* 

Monitor Report Columns 

PUPIL*-Ail pupils who are or have been mCTibers of the reading class 
are listed here* Pupils no longer in the class remain on the 
list because their previous monitor scores contribute to class 
averages » 

TEST FORM«-This column shows which test form the pupil took* There 
are 24 Achievement Monitors, four each at b±k levels with 
overlapping difficulty ranges* A double asterisk here indicates 
that the pupil did not take the test. 

PASSAGE SCORE-'-The six columns under this heading show the pupil's 

score on each passage of his achlevCTient monitor* Only the level 
1 forms (llj 12, 13^ and 14) have six passagesi all others have 
five* The passing score for a passage is 75%* A series of -l^s 
In the passage score columns Indicates absence from the test or 
incomplete test data* 

NEXT TA— The s and *^s In this column mark pupils who are to be 

given higher- or lower-level test forms at the next test admlnis^ 
tration. The general rule Is that a pupil will take a lower level 
form if he failed all passages and a higher level form If he 
passed all passages or all but one passage* 

MONITOR SCORE--The Monitor Score, based on passage perfontiancep is a 
difficulty level from I to 26, Indicative of the readability of 
the material the pupil can literally comprehend* The higher the 
monitor score, the more difficult the material* (How monitor 
scores are determined from passage scores is described In detail 
below*) Double asterisks in Monitor Score columns indicate 
absence* 

The Average Monitor Score for the class is the arithmetic mean 
of the individual monitor scores* 

The Monitor Report gives Monitor Scores and Average Monitor Scores 
for the most recent test administration and all preceding ainlnis- 
trationSy allowing progress in literal comprehension to be followed 
over timet 



Determination of Monitor Scores 

As fieted above, the monitor Brnxm is deteCTdned by the pupil's passage 
i€orei# If a pupil performs consistenclyt passing each successive passage 
en his achievement monitor until the passages become too difficult for him, 
hii^ monitor score will be the difficulty level of the last passage on i«?hlch 
he scored 75% or better. If, however, the pupil fluctuates, passing sot© 
passages and failing others, his monitor score will be the difficulty level 
of the second most difficult passage passed. If the pupil passes only one 
passage or falls all of the passages, his monitor score will be the difficulty 
level of the first (easiest) passage* 



To danonstrate, 
form 41 appear below. 


passage 


and monitor scores for four 


pupils 


taking 




Pupil 


Mil) 


Faisage 
(Difficulty level 
2(13) 3(15) 


Scores 
in pan 
4(17) 


snthesas) 
5(19) 


6C*) 


Monitor 
Score 




Susan Jones 


100 


100 


83 


83 


67 


0 


17 




Tom anlth 


100 


67 


50 


83 


83 


0 


17 




Terri Watson 


67 


83 


50 


83 


67 


0 


13 




ia.m Young 


50 


67 


. 33 


83 


50 


0 


11 





*No passage 6 on Fom 41, 

Susan Jones was consistent* She passed the first four 
passages and failed the last# Her monitor score Is therefore 
17, the difficulty level of the fourth passage. 

Tom Smith was inconsistent* He passed the first passage, 
failed the next two, and passed the fourth and fifth. His 
TOOnltor score is also 17, the difficulty level of the second 
most difficult passage on which he had a passing score. 

Tsrrl Watson failed, passed, failed, and passed again 
before falling the last passage. Her monitor score is 13* 

Kim Young had a passing score on only one passage, the 
fourth* Her monitor score is ll# 



The attached chart shows the difficulty levels of the passages on 
each of the test forms. It should be used in relating monitor scores to 
passage scores and analyzing pupil performances. 
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Literal Comprehgnsion Achlevenent Monitors 
Diffieulfcy Levels of Passages by F©m 
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Form 11 


Form 12 




Form 13 


Form 14 
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Form 21 


Form 22 




Form 23 


Form 24 
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4 
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4 




2 


6 


7 
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7 

/ 


3 


B 


9 




A 
w 


Q 
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4 


10 


11 




10 




5 


12 


13 
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Form 31 


Form 32 




Form 33 


Form 34 
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A 
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2 


9 
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11 
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11 
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13 
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Form 42 




Form 43 


Form 44 
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1 i 


2 


19 
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1 0 
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14 
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1 ^ 




1 4 




A 
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1 7 






1 ^ 

17 
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Level 










Form 51 


Fo™ 52 




Form 53 


Form 54 


1 


13 






1 % 


1 4 


2 


15 


16 




1 «i 


1 ^ 


3 


17 


IS 




1 7 


1 A 


4 


19 


20 




IQ 




5 


21 


22 




21 


22 






Level 


6 








Form 61 


Form 62 




Form 63 


Form 64 


1 


17 


18 




17 


18 


2 


19 


20 




19 


20 


3 


21 


22 




21 , 


22 


4 


23 


24 




23 


24 


5 


25 


26 




25 


26 
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RULES FOR CONSTRUCTING PARAFHIUSE 

iTiMS FOR wm AcmEwmm monitors 



Passage Saltatidn 

A* Dttarmlna range of difficulty for teat foms. • 

1, Identify eaah difficulty level in the Eeacttng/^terature 
HCC Exarciaes from which paaeagia will be dravm. 

2* Draw randomly the requiBite number of exereisea at 
each difficulty level* 

3. Replace dtleted worda in blanka in each MCC exercise 
dram* 

Paraphrasing Selected Exerci at Passage ^ 

A. Number each sentence in every exercise pasaaga* 

1. In paaMgas with compound aentances, number each main clause. 

2. In passages with complex sentences, numbar each main clauaat 
aubordinata clausat and long modifying phrase - 

B* Pairaphrasa each nwibared sentance or clauaa* 

1. If poaaiblei replace all aubstantiva words (noims, verba, 
modifiersJ) with ^onyms^ (i*e*, aquivalant n^rda or 
plwaaas)* 

a# Consult when necea^ry a dictioraryi theaauruSf or 
dictionary of synonyma* 

b. Cenault other ralavant referance words aa naceaaary* 

2© Propar nouns and pronoTOa often cannot ba paraptoased. 

3. Auxiliary verba and tha varb to be cannot always ba paraphrased 

km If posaiblei paraphraaa vocabtaary rfiould not axaaed thB 
vocabulary lavd- of the passage (as detarmined by difficulty 
lavel ) • 

a* Consult Harris and Jacobaon, 1972, when naaeasary* 

b. Consult Carrollf Davies, and Richman» 1971 t when necessary. 

5. Retain meaning of arip.nal aei^tance (l*e«, vocabulary and 

syntax of paraphrase ahould not involve aiff^ificant alteration 
of the literal meaning of tha original aentence). 



Rules for paraphrasing are based on AnderionLi (1972) daflnition of 
paraphraia* 
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C. Flexibility in the writing of paraphrases is illustratad belowi 

1. A paraphrase doas not have to have the exact niwiber of words 
as the original sentenee; it may be sll^tly longer or rfiorter* 

!• Syntax may br altered in vari qub ways. 

a« Order of elausea or phraaea may be changed as long ae 
literal meaung is retained « ' ^ ^ - ^ 

hm Voiea of verbs may be changed (e^g*« active to passive), 

c# Phrases may replaaa single words (and vice versa)* 

III, Writing Items for Paraphraaed Passaged 

A* Write WH^detail items on each paraphraiad sentence, clausei or 
phrase* Adhere as much as possible to the following rtilesl 

1. Write clear I concise, questions in colloquial Qiglish, changing 
the wording of the paraphrase as little as possible # (Excep* 
tioni replace pronouns with their referents*) 

2. Be^n each question with tlm appropriate detail word 
(e*g«| how, what, when, where, etc*)* 

3. Avoid writing inferential Vffl-detail items (a*g,, do not write 
a '•why" item imless the causal relatlon^ip is either explicit 
or clearly implied in the text)* 

km Write as many Vffl^detail items as possible for each paraphrase* 

5. Try to write as least two Vffl-detail items for each paraphrase 
Hotel Requl'rement for test forms was six WH-detail items/ 
passage* Passages are very ^ort (50-80 words )*6 

B* Write three distractors for each item (i.e., four responses, 
including distractors and correct re^onse)* 

1, Write only granmatlcally and semantically plausible 
distraotors. 

2. Write parallel distraators when possible* 

3* Write distractors that closely matdi the correct 
response in number of words* 

km Avoid writing re^onae arrays in which the correct 
response charaateristically stands out because of its 
brevity, length, or syntax* 

. 3m Write no distractors that could be correct in the context 
' of the passage. 
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6. Write distractora that art appropriate to the difficulty 
level of the passage (see II • fi» 4, above)* 



IV 0 Aroblema aad Re^onees 
A« Paraphrases 

1. Net evei^ sentenae yields an adequate paraphraaef For 
example, voeabid.ary levels i wiqueness of voeabtd.ary or 
atruoturet and other faators may make paraphraaing diffi* 
CiAt* 

2. When santenaae whiah aannot bt aaeeptably paraphraaed 
rasxilt in passages which do not yield the raquiaite 
number of itamss seleat another paesaga randomly from 
the relevant diffioulty levels? 

B. Items 

1« Vban item atama aontain aubetantive worda verbatim 
from tha paasagai make mv^m aorreat rasponse is not 
verbatim (i.e., do not write verbatim WH* detail 
items)* 

2* Whan a aorreat rasponaa is verbatim, WiJca aura that 
soma distraatora are alao verbatim to diminidi the 
poaatbility of ortho^aphia matching. 

5* When a correct rasponaa ia partially verbatim 

(a*0*t this ecaurp oaaaaionally in jlongar rasponses) t 
make sura at least one distract or aontaina the varbatim 
tlament which appaars in tha aorract response (to diminish 
orthop'aphia matahing)* 
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FdDtnotts 



Extraeted from eontext, subordinate elauses and sora© phraees may be 
paraptoaaed as mnin clausaa or sentances^ acampl©! "But even [a liar ^ a 
indention] , being an empty thing that offers no hold . . .''is paraphrasad 
as "a prevaricator's fiation is a vaeuoua thing that provides no handla" 
for a wh'itam as fallovel '%^t kind of thing is a pravariaator *s fiction?" 

%otal An alternate version of a santanoa, olausef or phrase vAioh 
••maana" what another santenoef clauae, or phraaa "maana" ia not neoassarily 
a paraphraae according to the rules here praaentad* Saying a thing in 
anothar way is not always equivalent to paraphrasing by these ri^es* 

Such a aituation occurs on occasion when a reviewer is diasatiafied 
with an item stem (or stem plua responae) and rawritas the item to make 
it aound batter or to avoid haavineasi awkwartoassi wordiness, etc**-but 
without first writing a new paraphraae or withQUt t^ing the ori^nal 
paraphraae into consideration. The rewritten item, conaidered out of 
context, vdll often aound or lo^ batter, but it will often no longer ba 
an item baaed on an acceptable paraphrase. 

A similar problem arises when an item is rewritten but is no longer 
a WH*detail item. 

Modifiers include adjectives and adverbs, not articlea or detarminers. 

^Superordinate terms are not necessarily acceptable synonyms 
(e.g., dog is not neceaaarily an acceptable s^onym for Siberian wolf-hound). 

5 

See Rulea for Constructing WH*Detail Items, on file with BSGR. 

^AveragejQumber of WH-datail items written for each paasage was more 
thmn tan, of which mix were selected. Criteria for selection were quality 
(e.g., absence of awkwardness and turgidity) and freedom from mutual cueing i 
defined as a stem giving away a re^onaa to another stem. In the following, 
for example, stem A cues the answer to stem Bi "A. When did the fuel drums 
burst into flame?" "B. Vtoat bwst into flame?" 

*^Fewer than tan per cent of the passages from the original sample had 
to be replaced. 
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